Polyglot: Automatic Extraction of Protocol Format using Dynamic Binary Analysis

Protocol reverse engineering, the process of extracting the application-level protocol used by an implementation, without access to the protocol specification, is important for many network security applications. Recent work [17] has proposed protocol reverse engineering by using clustering on network traces, but has several significant limitations. In this paper we propose a new approach to extract the protocol format using program binaries. Our approach, shadowing, uses dynamic analysis and is based on a unique intuition— the way that an implementation of the protocol processes the received application data reveals a wealth of information about the protocol format. We have implemented our approach in a system called Polyglot and evaluated it extensively using real-world implementations of five different protocols: DNS, HTTP, IRC, Samba and ICQ. Our results show that we can extract more accurate message format than previous work, and that the minimal differences of our results with respect to the protocol specification are usually due to different implementations handling fields in different ways. Finding such differences between implementations is an added benefit, as they are important for problems such as fingerprint generation, fuzzing, and error detection.